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ABSTRACT 



Although many difficulties are encountered in attempts to identify and use 
information about the outcomes of higher education, the pressures for under- 
standing the results 'of higher education are such that, we can no longer wait 
for the' perfect solution; we must act- now. This Statement depicts the current 
climate in which higher education decision makers find themselves, and it sets 
the stage for what is discussed in this document. 

Many things need to be considered in approaching the problems associated with 
the development and use of outcome information. One important consideration 
is to determine the uses to be made of the information. Will it be used for 
planning and management decisions, or will it be used for explaining and just- 
ifying programs and budgets? A second major consideration is to recognize 
that in order to understand the outcoroes of higher education it is critical 
to take into account the different components of higher education O.e., 
resources, educational processes, etc.) and their relationships to outcome 
measures. Another consideration is that, while research and development efforts 
are needed to help solve the outcojues information problem, much has been done 
in the way of measuring and analyzing higher education outcomes. An important 
step is for decision makers and researchers to get together to identify what 
outcome information is available and can be used, what the research and 
development priorities should be, and how multiple efforts to define and 
address the problem can be effectively coordinated. 



An initial effort of the National Center for Higher Education Management 
Systems at WICHE has been the development of an inventory of higher education 
outcome variables and measures. The intent of the inventory is to assist 
different kinds and levels of decision makers in communicating the desired 
and actual outcomes of instructional as well as research and public service 
activities. Two guidelines for using the inventory have been developed for 
(1) translating broad, vague goals into measurable terms and (2) developing 
priority lists of desired outcomes that can aid in choosing among alternative, 
courses of action. These guidelines and the inventory are contained in this 
document. 
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PART I 
INTRODUCTION 



Will postsecondary- education respond creatively to change and social trends 
by adapting and by influencing, or will its responses take the form of 
entrenchment or reprisal as a reaction to threats and crises? The question 
is crucial, and it is being asked today more and more frequently by 
observers of American higher education. The adaptability and responsiveness 
of our colleges and universities to the needs of individuals and society have 
long been the concern of planners and decision makers in higher education. 
Today, however, the pressures coming from outside the institution— from 
legislators, taxpayers, employers, parents, new applicants and others— as 
well as pressures being exerted from within— from students, faculty, and 
administrators— suggest that now more than ever before institutions of 
higher education need to reexamine the results and benefits of their programs 
from the perspective of the complex and dynamic issues confronting higher 
education and to plan accordingly. 

While college and university planners and decision makers are well aware of 
the issues confronting their institutions, they are beginning to recognize 
that sound directions cannot be evolved by concentrating solely on historical 
cost data. Likewise, they are aware that the detailed analysis of demographic 
data and institutional resources does not provide a final answer for making 
decisions and laying plans that will help the institution effectively adapt 
and respond to the needs of students, the local community, the state, and the 
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nation. As a result, decision makers are recognizing that for planning to 
be effective, it must not only be based on information about inputs, 
activities, and costs, but must also be oriented to making the outcomes of 
higher education relevant to the present and future needs of individuals 
and society. In short, there has emerged an understanding of the necessity 
for an "out come -oriented" approach to planning that is based on information 
about the results and benefits of an institution's programs rather than on 
information based exclusively on what goes into such programs and how those 
programs are operated. 

While most individuals concerned with higher education recognize the .need 
and urgency to utilize outcome information for purposes of program planning, 
management, and evaluation and also to support and justify long-range plans 
and budgets, they are quick to point out the complexity of the problem of 
identifying and" measuring the outcomes of higher education and incorporating 
this information in the planning and budgeting processes. 

One major difficulty is that few explicit measures of program effectiveness 
. are currently available. Nor has much yet been done to show the links 
between resources and activities used and the attainment of desired outcomes 
even when these desired outcomes can be quantified. It is much easier to 
see whether a plan has been accomplished in terms of activity or resource 
measures (e.g., expenditures, student/faculty ratios, enrollment levels) 
than in terms of educational outcomes. 



A second problem often cited is that even when information about outcomes 
is available, it is difficult to use since the techniques for analyzing and 
interpreting these data are largely undeveloped. For example, given all 
of the variables that potentially affect a particular outcome, it is 
virtually impossible in many instances to determine the cause-and-effect 
relationships. A further complexity results because many programs have 
joint outcomes. For example, a research project may contribute to student 
intellectual development in addition to producing research results. 

A third major difficulty is that most planners and decision makers simply 
have a hard time translating their goals into terms of the specific outcomes 
desired. Traditionally, goal setting is one of the first steps in the 
planning process; however, once the goals are stated, too often they remain in 
general, nonoperational terms. Because the goals lack translation into 
specific, measurable descriptions, planners and decision makers have trouble 
utilizing them in selecting the optimal, or even promising, courses of action 
and in evaluating the implemented programs. 

Finally, the use of outcome data is often thwarted by the fear of potential 
misuses. One aspect of this fear is that the data will not portray an accurate 
picture of the actual benefits derived from the institution and its programs. 
A second and perhaps more basic concern is uncertainty about the ultimate 
findings and the actions that will be taken by persons in positions of 
control. This latter concern is based on the fear that the evaluation 



process will not adequately take into account those outconies and benefits 
that are nonquantifiable and those inputs and goals that are unique to a 
given institution or program. 



PART II 

A FRAMEWORK FOR UNDERSTANDING THE USE OF 
HIGHER EDUCATION OUTCOME INFORMATION 



The complexity of identifying, measuring, and evaluating the outcomes of 
higher education is obvious and overwhelming; yet, it is becoming increasingly 
clear that all segments of higher education must learn to deal with these 
problems in the near future. 



Frederick E. Balderston (1970) quite accurately described the situation we 
now face in a paper entitled "Thinking About the Outputs of Higher Education": 



We have bumped hard into the question of outputs and their measurement 
because, among other things, we are seeking now to link the resources 
used to the results achieved— In other words, to link inputs with out- 
puts. It turns out that in the long history of concern about the 
processes and activities of education, we have achieved only a very 
imperfect grasp of the nature of its results. Now we are having to 
tackle the problems of output definition and measurement under forced 
draft, for higher education has come to the front of public attention 
both as a major social problem and as a major contributor to social 

change and economic development The job we have to do is urgent, 

important and controversial. If we had time, we might do well to 
sympathize with ourselves for taking it on. 



While few doubt the need, pressures, and complexities of using outcome information 
in laying plans, making decisions, and justifying budgets, many questions 
remain to be answered. For example, what outcome inforniation is going to 
be utilized? For what purpose? What is the relationship between the outcomes 



of higher education programs and other program measures?* What is the degree of 
congruence between desired outcomes and actual outcomes? What is the state of the 
art in college impact research and how can this be capitalized upon for short-term 
implementations and built upon for long-term improvements? Finally, what are 
the ground rules and procedures for collecting, analyzing, and utilizing outcome 
information? 

The purposes of this section are to address these questions, to identify the 
knowledge and capabilities that exist, and thereby to establish a framework 
for utilizing outcomes in higher education planning and decision making. 

The procedures one chooses for utilizing outcome information and the kinds of 
data and comparisons that he will need are dependent upon the use to be made 
of that information. While there are many purposes for focusing on outcomes, 
at least two can be readily identified. First, outcome information can be 
employed within an institution to aid the program planning, budgeting, and 
managing processes, or it can be exchanged with other institutions for these 
same purposes. Second, outcome information can be used to support and justify 
budgets, expenditures, and long-range plans. These processes are of major 
importance to funding agencies and constituents. The latter use is generating 



*Currently, NCHEMS has undertaken a project entitled Program Measures for 
developing categories of information tfiat quantitatively describe programs 
in higher education. Collectively these information categories have been 
defined as "program measures" and are grouped as follows: Resource Measures 
(personnel, facilities, equipment). Financial Measures (revenues, expenditures). 
Target and Beneficiary Group Measures (those groups who are directly served 
and those who ultimately benefit). Activity Measures (operations), and Outcome 
Measures (products, events, and conditions). 
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much of the urgency and pressure for outcome information, but over time 
the first will lead to sounder directions for higher education. As a 
result it will provide a better response to calls to accountability by various 
consti tuencies. 

To carry out these purposes, decision makers need to choose among alternatives, 
and therefore a process of comparison is required. The nature of this comparison, 
however, can be quite different if, for example, the comparison is of different 
programs rather than of the same program. across time. Thus, another dimension 
to be understood in using outcome information depends on the .type of comparisons 
that may be involved. Different bases for comparison could be appropriate for 
the data assembled and the procedures used. For example, institutions or agencies 
may compare their performance with some goal they t.ave established, with national 
norms, with data from some selected groups of comparable institutions or programs, 
or with historical program data. 

Equally important in the use of outcome information is an understanding of the 
different components of higher education and their relationships to outcome 
measures. Figure 1 presents a simplified model of higher education that shows 
the relationships between the major components. 
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Figure 1. The Relationship Between the Major Components of 
Higher Education. 



While the isolated effect of institutional resources, student inputs, the 
institutional environment, or educational processes on educational out- 
comes for a given time period may be relatively straightforward, all of 
these components are interrelated over time. This interrelationship becomes 
relevant in the context of planning and decision making, and it also points 
out the importance of using both immediate and long-term observations to 
gain a better understanding of measures that characterize higher education. 
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For example, student inputs can directly affect student outcomes, and either 
student inputs or institutional resources can change the. institutional environ- 
ment and the educational process. These changes may, in turn, affect outcomes. 
Introducing a special educational opportunities program by admitting and sup- 
porting underfunded students will change the socioeconomic mix and possibly the 
racial mix on campus. These changes will likely affect certain values and 
attitudes of the entire student body. Such a program may require special 
courses, counseling, and cutoring which may represent a change in the 
educational process and require a different kind and amount of institutional 
resources. 

In addition, the impact of higher education outcomes can over time change 
the institutional environment. For example, certain research outcomes may 
eventually attract or develop a different kind of faculty and alter curricular 
emphasis in terms of the graduate/undergraduate ratio. Educational outcomes 
may also affect inputs in the future, both in terms of the resources that are 
made available and the kinds of students that seek admission. For example, 
alumni may encourage f)otential students to enroll or not to enroll. Impacts 
on job markets can affect the amount of dollars that the institution receives 
over time. 

Educational processes carried out in one period can also affect the institutional 
environment in future periods. For example, changes in curricula or instructional 
methods may attract a different kind of student or faculty member in future 
time periods and cause a change in the environment of the institution. 



One can easily imagine other multiple interrelationships of the major components 
of higher education and recognize the influence each has upon the others in a 
chain-reaction process over time. In short, the measurement of outcomes and 
the utilization of outcome data will likely be misleading if considered 
independent of measures of other components of higher education. 

Measuring the outcomes of higher education will remain at the heart of the 
"outcome problem" for some time, perhaps for all time. But a good deal of 
research effort already has been directed toward measuring the outcomes of 
higher education, and it should be possible to capitalize on many of these results. 
(See, for example, Feldman and Newcomb, 1969.) Although it is not the purpose 
of this document to review comprehensively the state of the art in outcome measure- 
ment and college impact research. Table 1 attempts to capsulize the nature of 
the work that has been accomplished and the potentials for future research. 

The first category identified in Table 1, "College Impact on Students," focuses 
on changes in individual student sk'lls, abilities, and characteristics 
resulting from the educational experience. Most college impact research has 
been in this category, and much information about such outcomes as general 
knowledge and vocational preparation can be derived from the studies and the 
measures now available. These studies generally have used all or parts of 
the model shown in Figure 1 and employ questionnaires or test instruments 
to gather data on students or recent graduates over a relatively short 
time. 
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TABLE 1 
COLLEGE IMPACT RESEARCH 





Focus of the Research 


Outcomes Examined 


— 


-Short Time Horizons 


Knowledge 


COLLEGE IMPACT ON 


-Macro Studies 


Skills 


STUDENTS 


-Model of Student Inputs, 
Envi ronment/Process , 
Student Outputs 


Values and Attitudes 
Competence 




-Utilizes Primarily Test 
and Questionnaire Data 


Vocational Preparation 




-Long Time Horizons 


Earning Power 


COLLEGE IMPACT ON 
GRADUATES 


-Macro Studies 

-Historical Time Series 
Data (Census, etc.) 


Redistribution of 
Wealth 

Rate of Return 




-Focuses on Economic and 
Labor Factors 


Socioeconomic Mobility 



Employment 

Intergenerational Effects 



COLLEGE IMPACT ON 
SOCIETY 



-Medium-Range Horizons 
(1 to 10 years) 

-MacT udies 

-Eirplrical Data from 
Industry, Government, 
Social Agencies, etc. 

-Focuses on Both Social 
and Economic Factors 



GNP Factors 
Crime Rate 

Manpower Supply/Demand 

High Risk/High Cost 
Subpopulations (Welfai^, 
Mental Health, etc.) 

Social Participation 
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The second category of studies has been called "College Impact on Graduates" 
(with some misgiving because the title does not adequately describe the nature 
of the available research of this type). Most of the work in this area has 
been accomplished by economists, and it has, in general, focused on the long- 
term and in certain cases the intergenerational effects of higher education 
and has typically utilized global, historical data. A number of studies have 
looked at the long-term earnings of college graduates, their socioeconomic 
mobility, etc. This research could be replicated for particular sectors and 
institutions of higher education and utilized in planning and management. 

The third type of measurement study has somewhat arbitrarily been titled 
"College Impact on Society." While it can be argued that many of the so- 
called "College Impact on Graduates" studies in the second category have 
implications for society as well as individuals, attention should also be 
given to shorter-term effects on production, human vyelfare, and the other 
kinds of outcomes that are described in this third category of college 
impact research. Such studies would undoubtedly involve shorter time horizons 
and would draw upon data from employers, social agencies, and various other 
sources. 

Certain characteristics and future research requirements are common to all 
three impact areas. First, because of rapid changes in higher education, 
impact research must be continuous. Changes in higher education over time 
can easily invalidate relationships that were uncovered years earlier. Also, 
many of the measurements and relationships are remote in terms of time. A 
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change in the educational process may precede by a number of years a correspond- 
ing change in a resulting outcome measure. The same may be true of resource 
measures and environmental measures. 

Thus, many outcomes of higher education can be thought of as investments; that is, 
some value— an increase in knowledge, for example— is added over time. Other 
outcomes are more like consumer goods and are evali'uced in terms of iitmediate 
reactions, like satisfaction gained by some new discovery or the joy of learning. 
These differences obviously imply the need for different measurement methodologies. 
Perhaps sets of measures can be combined into indices similar to the Gross 
National Product, the Dow Jones Average, and the Cost-of-Living Index. Such 
indices have important meaning for investors, economists, and governmental 
policy makers. The same could be true for htgher education in the future 
by employing these kinds of approaches. 

Manv difficulties and dilemmas compound the problem of measuring the outcomes 
of higher education. Most of these difficulties, however, center around 
interpretations of outcome measures. It is often confounding to attribute 
given outcomes to specific courses. For example, suppose a significant 
relationship is found between the number of credits taken in elective courses 
and the vocational advancement of graduates a number of years later. This 
observation is difficult to interpret. Did the elective courses make the dif- 
ference, or was some characteristic of students who take elective courses- 
such as a broad scope of interests— really the causal factor? Systematic 
research should ease the problem created by too many variables and interrelation- 
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ships to identify and study. Psrhaps a critical path through the maze can 
be defined and carried out in a well -coordinated effort. 

The need and urgency to begin internalizing and implementing outcome-oriented 
approaches have come down hard upon higher education, even though there are 
high risks and danger of damaging misuse of the meager information that is 
available. While our knowledge is incomplete and imperfect, it should not 
be underestimated. It is unrealistic to insist upon the ability to measure 
quantitatively all the results and benefits of higher education before taking 
the first step down the road in the direction of outcome-oriented planning 
and management. The nature of these problems indicates that a long-term, 
large-scale effort is the prospectus for a program in the area of the outcomes 
of higher education. Such a program needs to accomplish: 

1. The development of better outcome measures, and the identification 
of relationships between these measures. (This is primarily a 
research problem involving specific measurement studies and 
improving measurement technologies.) 

2. The establishment of linkages between outcome measu>^s and other 
measures of the components of higher education. 

3. Continual improvement of the conceptualization and understanding 
of the outcomes assessment problem in higher education. 
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4. The development of s taxonomy of higher education outcome variables 
that Identifies and; defines the associated outcome measures that 
are currently avalliWf^.'j, (This product must have a very pragmatic 
orientation.) 

5. The development of guidelines and procedures for utilizing 
outcome measures and assessment methodologies that have short- 
term pj^yoffs. and can be built upon for longer-term Improvements. 

6. Finally, the promotion of the use of outcome Information In the 
planning and management processes In all sectors and levels of 
higher education. 

Achievement of these results will require considerable resources over a long 
period of time and will Involve parallel activities In research. Implementation, 
and coordination. These achievements will be accomplished only If the decision 
makers and policy setters on the firing line take the lead by giving direction 
and coordination to the overall effort. 
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PART III 

THE INVENTORY OF HIGHER EDUCATION 
OUTCOME VARIABLES AND MEASURES: AN OVERVIEW 



The Outcomes of Higher Education project of the National Center for Higher 
Education Magagement Systems at WICHE hopes to make significant contributions 
to solving the problems associated with identifying and using outcome infor- 
mation in planning and management. NCHEMS's first effort has been an attempt 
to develop an inventory of possible outcome variables with suggestions for 
their measurement. The sections that follow describe the inventory and its 
use. 

Development of the Inventory 

One major problem associated with incorporating the outcomes of higher 
education into planning and decision making processes has been the lack of 
a "common outcomes language" necessary for comnunicating and understanding 
the outcomes and benefits of higher education programs. To some degree 
this situation parallels the difficulties biologists faced prior to the 
development of Linnaeus 's taxonoiny, which provided a common language or 
inventory for identifying and categorizing the various hierarchies of 
living organisms. Once the taxonony was completed, however, biologists 
were in a better position to identify, measure, and analyze the characteristics 
and changes of the various species. Consequently, their knowledge about 
plant and animal organisms increased, and their comnuni cation about these 
organisms with other scientists improved. 
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Similarly, higher education has difficulty communicating about benefits or 
results. Barriers are encountered when attempts are made to translate 
goal statements into terms of program outcomes, and program comparisons 
are thwarted because structures, definitions, and measures are lacking. While 
developing a communication base for higher education outcomes is by no means 
a total or final solution, it is a necessary step. Recognition of the 
potential benefits of such a communi caption base has led NCHEMS to develop 
the Inventory of Higher Education Outcome Variables and Measures , This 
inventory lists and describes various outcomes of higher education and 
suggests potential measures or proxy measures of those outcomes,* 

Basic to the inventory are two criteria- The first criterion is that the 
inventories must be of service to as many kinds and levels of planners 
and decision makers in higher education as possible. For example, they 
should aid students and parents in making better decisions about ^hich 



*NCHEMS has also developed an Inventory of Institutional Environment 
Variables, and Measures that includes various combinations of resource measures, 
financial measures, activity measures, and target and beneficiary group meas- 
ures. The development of this inventory recognizes that institutions do 
design programs and allocate resources to produce certain important environ- 
mental outcomes. For example, institutions of higher education create pro- 
grams and allocate resources to develop certain types of facilities that will 
promote a unique intellectual and/or social atmosphere. While the creation 
of this unique atmosphere may be a desired outcome, it is pursued with the 
belief that it will eventually contribute to better student learning, better 
research, or better service for the students and the community in general. In 
addition, environmental measures can be utilized as meaningful proxies for out- 
comes, if the outcomes cannot be measured directly. For example, the number of 
library volumes acquired per student may serve as a proxy measure of student 
growth and development. 

For your information, drafts of this inventory are available on request. 
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institutions and programs will provide the most meaningful and appropriate 
educational experiences. They should help institutional administrators and 
program managers account for the educational resources allocated and utilized 
in terms of the outcomes and benefits produced and the goals attained. Finally, 
they should provide legislators and statewide coordinating agencies with a 
better understanding of the intended as well as the unintended consequences of 
higher education. 

The second criterion employed in developing the inventory is that it must 
provide a relatively complete characterization of an institution's programs. 
The variables listed in the outcomes inventory should include not only 
academic and instructional outcomes, but research and community service out- 
comes as well. 

The inventory has been developed to include comprehensive lists of the 
outcome variables related to higher education programs and institutions. 
For the purpose of clarification, a variable in the context of the inventory 
is defined as some entity or quality capable of assuming one of a number of 
quantitative or qualitative values. For each outcome variable the 
inventory presents a definition and/or pertinent description, and it suggests 
a list of potential measures that can provide the appropriate evidence or 
necessary data for assessing the designated variables. 

The current outcomes inventory incorporates reactions and suggestions from 
many individuals concerned with higher education. The following major 
categories define the structure of the inventory. 
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Section 1.0: Student Growth and Development Outcome Variables 

1.1.0: Knowledge and Skills Development 

1.2.0: Social Development 

1.3.0: Personal Development 

1.4.0: Career Development 
Section 2.0: Development of New Knowledge and Art Forms Outcome Variables 
Section 3.0: Community Development and Service Outcome Variables 

3.1.0: Community Development 

3.2.0: Community Service 

3.3.0: Longer Term Community Effects 

Characteristics of the Inventory 

The outcomes inventory can be further described by identifying certain key 
characteristics and limitations. 

Comprehensiveness: While every attempt has been made to develop 
a comprehensive list of variables, it is highly probable that 
certain important outcome variables have been overlooked. Or, 
more likely, in certain cases the descriptions may be interpreted 
to exclude elements that they are intended to encompass. 

It is also recognized that different individuals, institutions, 
and agencies will establish different subsets of the variables 
they view as relevant. Such lists will undoubtedly eliminate 
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certain variables, which will simply emphasize the fact that 
different individuals, institutions, and agencies have unique 
sets of objectives. 

Disaggregation : If an attempt is made to map the inventory 
onto an institution's program structure, it may become apparent 
that incongruities exist at different levels of aggregation. For 
example, it is very difficult to associate many of the outcome 
variables, particularly those in the area of student values and 
attitudes, with any program classification below the entire campus 
except on a very arbitrary basis. A major reason for this aggrega- 
tion problem is that higher education programs often produce joint 
outcomes. For example, a program in political science potentially 
affects students in terms of their "political" values and attitudes. 
Similarly, a program in history and sociology can also affect 
"political" values and attitudes. Consequently, attributing any 
change in students* "political" values and attitudes to a particular 
program or course is extremely difficult. 

Redundancy : While developing a list of mutually exclusive outcome 
variables has been a key concern in the development of the 
inventory, the overlaps between variables in the inventory have 
not been entirely eliminated. For example, student values and 
attitudes toward "change and stability" are likely to intersect 
with "political" values and attitudes. Similarly, "vocational 
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preparation" characteristically intersects with "general 
knowledge" and "communication skills." 

Neutral Scale: The variables' definitions and/or descriptions 
are not intended to connote value judgments. Efforts h^^ve 
been made to eliminate the use of such value-laden terms as 
"increase," "gain," and "benefit." Each user of the inventory 
is expected to view the variable descriptions as a neutral scale, 
to which he can attach his own unique values in terms of his pre- 
ferred evaluation standards or his desired levels of performance. 
For example, one institution may want to increase the importance 
its students attach to "socioeconomic aspirations," while 
another institution may desire to decrease the degree of emphasis 
placed on this variable. It is also r-ecognized that an implied 
value judgment is built into the inventory by virtue of the 
level of aggregation utilized. That is, since "communication 
skill" is listed and mathematical skill is not, the unintended 
implication may be that communication is more important than 
mathematics. However, every attempt has been made to maintain 
a consistent level of aggregation throughout the inventory. 

Measures ; Developing a comprehensive list of outcome 
measures is a large scale task, and it should be clear that the 
suggested measures are not all-inclusive or, for that matter, they 
may not be the best available. Thus, each user of the inventory 
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should strive to identify or develop additional measures or proxy 
measures to gain as much information as possible about the outcome 
variables he is interested in assessing. The criteria for suggest- 
ing measures are based on judgments of their significance and 
practicality, primarily relative to data availability. Studies 
to determine the relationships between measures and their value, 
practicality, and interpretation remain to be accomplished. 
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PART IV 
USING THE INVENTORY 



Outcomes can be an integral part of higher educational planning and decision 
making, even though in many cases information is incomplete or quantitative 
measures are not available. The following general suggestions for using the 
outcomes inventory are presented with this belief in mind. 

Translating Goals into Measurable Terms 

The major shortcoming of most goal statements— that they are nonoperational 
and unmeasurable— could be alleviated through the structure and definitions 
—provided in the inventory and the application of the following guidelines: 

Guideline 1; Identifying the Desired Outcomes . The first task in 
translating goals into operational, measurable terms should begin 
by answering the following question for each goal: "Which outcome 
variables in the inventory are representative of the goal state- 
ment?" Consider the following goal: 

"To develop the career potential of each student." 

Clearly, this statement does not offer much information about specific 
outcomes or benefits that should result from programs implemented to 
attain it. Through the use of the outcomes inventory, however, tne 
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following "Student Growth and Development" outcome variables could be 
Identified as reflecting the actual meaning of the goal: 

"To develop the career potential of each student." 
General Knowledge 
Knowledge in Specialized Fields 
Application of Knowledge 
Vocational Preparation 

Obviously, there is a good chance for disagreement with this list of 
outcomes because it does not contain all of the outcomes related 
to the goal, or one might argue that other outcome variables in the 
inventory define the goal better. Since a prerequisite of the goal 
translation process is to identify a set of outcome variables for 
each statement that is as close as possible to consensus, some 
method that allows a pooling of individual judgments should be used. 
For example, survey techniques such as the DELPHI method (Uhl, 1971) 
or the Q-sort technique CStephenson, 1953) could be useful.* 

Guideline 2-:- Identifying the Outcome Measures. The next task 

in translating goal statements is establishing the means for quantifying 

the outcome variables. For each of the outcome variables, measures 



♦These techniques are designed to help in reaching consensus among persons 
who may have differences of opinion about some subject. 
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that will provide quantitative information necessary for evaluating 
the outcome should be selected from the inventory. If a representative 
outcome measure cannot be identified in the inventory, then it may 
be possible to substitute some other proxy measure. 

As Figure 2 shows, a master outcome list for a single goal includes a number 
of different levels of specificity. 



Goal 
Statement 



Outcome 
Variable 



Outcome 
Variable 



Outcome 
Variable 



— Outcome Measure 

— ;tcome Measure 
Outcome Measure 

— Outcome Measure 



Figure 2. A Diagram of the Levels of Outcome Specificity. 
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The primary potential misuse of the inventory at this step is the tendency 
to attach an inappropriately larger importance to those variables that are 
easiest to measure. Hartnett (1971, p. 13) points out that "such a situation 
is much like that of the proverbial tail wagging the dog.' It is important 
to recognize that certain consequences of higher education probably are 
not measurable and perhaps never will be. The following points need to 
be considered when outcome measures are selected: 

Will the measure be valid? Unless the measure is judged to be 
representative of the outcome being measured, it will have little 
meaning to those who are using the data to make a decision. 

How reliable is the measure? The consistency and reproducibility of 
a measure should always be considered to guard against variance due 
to random error. 

Where will the data come from? A measure will be of little use 
if no feasible data source is available or if appropriate data 
collection instruments and methodologies are lacking. 

How much time will it take to implement the measure? The time 
frame required to use a particular measurement technique should 
be considered in terms of feasibility and timeliness. 

How much will a particular measure cost? While certain types of 
data are desirable for evaluating programs, the costs of collecting 
those data must be analyzed at the outset. 
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Wm the measure constitute an "invasion of privacy"? Although 



certain measures are quite satisfactory for the purpose intended, 
they cannot be utilized if individuals or groups cannot be asked 
to supply the data needed or if they refuse to supply it. 



In summary, before implementing any measure, the adequacy of that measure 
(i.e., its relevance, reliability, validity, and ease of administration) 
should be well established. Every attempt has been made to apply these criteria 
in suggesting the measures listed in the inventory. 



Guideline 3; Determining the Basis for Comparison . The next step in 
translating goals involves Identifying the evaluation criteria or bases 

: against which comparisons can be made between actual and desired outcomes. 

|i Generally, there are three types of comparison that can be utilized. First, 

|. an actual outcome can be compared to some "absolute" standard or goal 

I that has been set for the specific level or degree of performance to be 

I achieved. Such standards usually specify the particular minimum and/or 

I 

%. maximum levels of performance to be achieved. It is also possible to use 

I a historical outcomes data base as a basis for comparison. Often it is 

t. of interest to know how some program has done relative to past performance. 

I This means establishing a data base that can be maintained and utilized for 
|i making evaluations over designated periods of time. A third possibility 

fe; is to use the performance of the comparable programs or the achievement of 

II other groups of students as the basis for comparison. 



I 
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Establishing a basis for comparison involves determining the "best test 
for success" relative to each desired outcome. Quite naturally these 
decisions provide the potential for great conflict. The outcomes inventory 
provides only indirect assistance in this area, since by design the out- 
come variables and measures are intented to be "neutral" in nature. 

However, in identifying the bases for comparison against which evaluative 
judgnents can be made, the following points should be kept in mind: 

The basis for comparison should be consistently and accurately 
measurable. 

The basis for comparison should relate to the specific situation in 
focus. It should be valid. 

The basis for comparison should be updated as dictated by the 
dynamics of the higher education environment. 
The basis for comparison should be acceptable to the groups 
affected by the comparison; again, consensus is important. 

To facilitate the goal translation process, a "goal translation recording 
form" similar to Table 2 wou*;a be helpful. This table Illustrates how an 
outcome variable such as "vocational preparation" might be analyzed. The 
following measurement-specific statements of desired outcomes have been 
transcribed from the information shown in the table: 
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The Mechanical Engineering program at Micro U will demonstrate 
effectiveness in its vocational preparation of students as evidenced 
by 90 percent of the program's graduates being employed within 90 
days after graduation. (A comparison based on an "absolute" standa^-d 
or goal.) 

The Mechanical Engineering program at Micro U will demonstrate 
effectiveness in its vocational preparation of students as 
evidenced by the average number of job offers of graduates 
being at least as great as the average first salaries of Mechanical 
Engineering graduates at Macro U. (A comparison based on the 
performance of another program.) 

The Mechanical Engineering program at Micro U will demonstrate 
effectiveness in its vocational preparation of students as 
evidenced by the class of 1965 achieving 5% more management 
positions five years after graduation than the class of 1964 
five years after graduation. (A comparison based on historical 
data. ) 

In practice these desired outcome statements would deserve further qualification; 
however, they are illustrative of the type of measurable goal statements necessary 
for effective outcome-oriented planning and decision making. The major 
advantages of such statements are: 
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They clearly communicate in measurable program outcome terms what is 
desired. 

They are much easier to compare because they are in the same 
language as the actual outcomes of programs, and the basis for 
comparison is explicitly stated. 

They help identify the kind of measuring instruments to utilize 
for data collection. _ 

Developing a Priority List of Outcomes 

"Which alternative program is most Ukely to give us the most return for our 
money and effort?" As is often the case, no one program alternative can be 
expected to satisfy all of the desired outcomes. As a result, in selecting 
programs and allocating resources, planners should constantly refer 
to the relative value of the desired outcomes spelled out in a "priority list" 
of outcomes. 

Guideline 1: Developing a Prlortty List of Desired Outcomes . Programs 
produce multiple outcomes, and different programs result in different 
kinds and levels of outcomes. Programnatic decisions must focus on the 
differences between various alternatives in terms of their importance 
relative to each other. As a result, the following points should be 
considered: 
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Once the outcome variables and measures have been identified, 
they should be ranked in order of importance or they should 
be assigned quantitative weights to reflect their relative 
values. 

Since different decision makers may value the outcomes quite 
differently, the techniques for reaching consensus again 
should be employed. 

Guideline 2; Applying the Priority List of Outcomes . Once constructed, 
the ranked or weighted priority list of outcomes can be used as a "yardstick" 
to evaluate the expected outcomes of the programs being considered. The 
end result will be an estimate of the benefits to be derived from the 
various program alternatives being analyzed. The example in Table 3 
illustrates how such a priority list of desired outcomes could be 
applied in program selection. 

TABLE 3 

A COMPARISON OF THREE ALTERNATIVE PROGRAMS 
USING A LIST OF RANKED DESIRED OUTCOMES 





Ranked 
Desired Outcome 




Expected Outcomes of 
Program Alternatives 

Ag Aj 


1 - 


Vocational Preparation 


0 


+ + 


2 - 


Application of Knowledge 


+ 


+ + 


3 - 


General Knowledge 


+ 


0 + 


4 - 


Political Attitudes 


0 


+ 0 



{+) = outcome expected ; (0) = outcome not expected 
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As the information in the table indicates, alternative will produce 
the most benefits. However, if the cost of implementing alternative A2 
is half the expected cost of alternative A^, then a decision will have 
to be made concerning whether the outcome, "general knowledge," is worth 
the difference. Although this example is greatly oversimplified, it does 
suggest the process involved. 

In Conclusion 

The few guidelines that have been presented are admittedly subjective, and in 
many instances will be difficult to apply systematically. As a result, many 
individuals will say that until the outcomes inventory and the guidelines 
are fully developed and we know all there is to know about their use and 
consequences, no attempts should be made to employ them in laying plans and 
making decisions. While NCHEMS is sensitive to the need for much caution, 
it also is aware that many ottier individuals are saying, "We recognize the 
potential limitations and misuses, but we cannot wait; we must act now." 
Consequently, as the inventory and the guidelines are examined and as efforts 
are applied to implement them, this experience will be utilized to improve 
the inventory and to develop more definitive guidelines for its use. It 
is, therefore, important that the planners and decision makers in higher 
education take the lead in this effort by effectively collecting and utiliz- 
ing the outcome data that are available and feeding back the results of these 
experiences to focus additional research. 
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In support of this effort, the NCHEMS Outcomes of Higher Education project has 
two objectives in mind: First, to coordinate a well-planned research effort 
so the problems associated with the inventory and with the use of outcome 
data in higher education planning and decision making will be better under- 
stood and, second, to meet the immediate needs of higher education by 
synthesizing and testing procedures and measures and making these available 
as soon as possible. 
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OUTLINE OF THE INVENTORY OF OUTCOME 
VARIABLES AND MEASURES 

1.0 Student Growth and Development 

1.1.0 Knowledge and Skills Development 
1.1.1.00 Knowledge Development 

1.1.1.01 General Knowledge 

1.1.1.02 Specialized Knowledge 
1.1.2.00 Skills Development 

1.1.2.01 Application of Knowledge Skills 

1.1.2.02 Critical Thinking and Reasoning Skills 

1.1.2.03 Creativity Skills 

1.1.2.04 Communication Skills 

1.1.2.05 Motor Skills 

1.1.3.00 Knowledge and Skills Attitudes, Values, and Beliefs 
1.1.3.01 Intellectual Disposition 

1.2.0 Social Development 
1.2.1.00 Social Skills 

1.2.1.01 Interpersonal Participation 

1.2.1.02 Leadership 

1.2.1.03 Citizenship 

1.2.2.00 Social Attitudes, Values, and Beliefs 

1.2.2.01 Political 

1.2.2.02 Racial /Ethnic 

1.2.2.03 Personal Ethics 

-^^39 



1.2.2.04 Social Conscience 

1.2.2.05 Socioeconomic Aspirations 

1.2.2.06 Cultural Interest 

1.3.0 Personal Development 
1.3.1.00 Student Health 

1.3.1.01 Physical Health 

1.3.1.02 Mental Health 

1.3.2.00 Student Personal Attitudes, Values, and 

1.3.2.01 Religious and Spiritual 

1.3.2.02 Change/Stability 

1.3.2.03 Self-Concept 

1.4.0 Career Development 

1.4.1.00 Career Preparation 

1.4.1.01 Academic Preparation 

1.4.1.02 Vocational Preparation 
1.4.2.00 Career Attitudes, Values, and Beliefs 

1.4.2.01 Achievement Orientation 

1.4.2.02 Educational Aspirations 

1.4.2.03 Educational Satisfaction 

1.4.2.04 Vocational Aspirations 
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Development of New Knowledge and Art Forms 
2.0.0.01 Discovery of New Knowledge 
2.0.0.02 Interpretation and Application of New Knowledge 
2.0.0.03 Reorganization of New Knowledge 

Community Development and Service 

3.1.0 Community Development 

3.1.0.01 Community Educational Development 
3.1.0.02 Faculty /Staff Educational Development 

3.2.0 Community Service 

3.2.0.01 Extension Services 
3.2.0.02 Personal Services 

3.2.0.03 Extramural Cultural and Recreational Services 
3.2.0.04 Financial Impact on the Community 

3.3.0 Longer Term Community Effects 
3.3.0.01 Social Impact 
3.3.0.02 Economic Impact 
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